Calculating noise estimates of a digital image using gradient analysis

ABSTRACT

A method of estimating the noise in a digital image for use in subsequent image processing, includes receiving a source digital image having a plurality of pixels; selecting a pixel of interest; providing a plurality of orientations for the pixel of interest; using gradient analysis on the source digital image and the plurality of orientations to select the most suitable orientation for estimating noise for the pixel of interest; using the selected orientation in the pixel of interest to determine a noise-free pixel estimate for the pixel of interest; and repeating for other pixels of interest and using the noise-free pixel estimates to calculate a noise characteristic value representing the noise estimate for the source digital image.

FIELD OF INVENTION

[0001] The present invention relates to providing noise estimates thatcan be used to remove noise from digital images.

BACKGROUND OF THE INVENTION

[0002] Some digital image processing applications designed to enhancethe appearance of digital images take explicit advantage of the noisecharacteristics associated with the digital images. For example, Keyeset al. in U.S. Pat. No. 6,118,906 describe a method of sharpeningdigital images which includes the steps of measuring the noisecomponents in the digital image with a noise estimation system togenerate noise estimates and sharpening the digital image with an imagesharpening system which uses the noise estimates. Similarly, digitalimaging applications have incorporated automatic noise estimationmethods for the purpose of reducing the noise in the processed digitalimages as in the method described by Anderson et al. in U.S. Pat. No.5,809,178.

[0003] In commonly-assigned U.S. Pat. No. 5,923,775, Snyder et al.disclose a method of image processing which includes a step ofestimating the noise characteristics of a digital image and using theestimates of the noise characteristics in conjunction with a noiseremoval system to reduce the amount of noise in the digital image. Themethod described by Snyder et al. is designed to work for individualdigital images and includes a multiple step process for the noisecharacteristics estimation procedure. A first residual signal is formedfrom the digital image obtained by applying a spatial filter. This firstresidual is analyzed to form a mask signal which determines what regionsof the digital image are more and less likely to contain image structurecontent. The last step includes forming a second residual signal andsampling the second residual signal in the image regions unlikely tocontain image structure as indicated by the first residual signal. Themethod taught by Snyder et al. requires the use of the mask signal toproduce accurate noise estimates due to the fact that the spatial filterused to calculate the second residual image does not fully filter theimage structure content.

[0004] It is desirable in any noise estimation method to obtain aresidual signal that is pure noise, with no image structure content.This will lead to more accurate estimation of the noise characteristicsin the image. Existing techniques suffer from the problem of imagestructure contamination in the residual signal used to estimate thenoise. In other words, the spatial filter that produces the residualsignal does not fully filter out image structure. The masking techniquecannot fully exclude image structure pixels from the residual signal.

SUMMARY OF THE INVENTION

[0005] It is an object of the present invention to provide a noiseestimates for a digital image that can be used in subsequent processingto enhance digital images.

[0006] It is a further object of the present invention to provide fornoise reduction and spatial sharpening processing of a digital imagethat uses the noise estimates.

[0007] These objects are achieved by a method of estimating the noise ina digital image for use in subsequent image processing, comprising thesteps of:

[0008] a) receiving a source digital image having a plurality of pixels;

[0009] b) selecting a pixel of interest;

[0010] c) providing a plurality of orientations for the pixel ofinterest;

[0011] d) using gradient analysis on the source digital image and theplurality of orientations to select the most suitable orientation forestimating noise for the pixel of interest;

[0012] e) using the selected orientation in the pixel of interest todetermine a noise-free pixel estimate for the pixel of interest; and

[0013] f) repeating steps b)-e) for other pixels of interest and usingthe noise-free pixel estimates to calculate a noise characteristic valuerepresenting the noise estimate for the source digital image.

[0014] It is an advantage of the present invention that by usinggradient analysis as a feature of a noise estimation spatial filter thata noise characteristic value representing an estimate of the magnitudeof noise present in the digital image can be computed and used to removenoise from the digital image. It is a further advantage of the presentinvention that the noise characteristic value can be used to enhance thespatial detail of a digital image while minimizing the amplification ofthe noise.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015]FIG. 1 is a functional block diagram of a digital image processingsystem suitable for practicing the present invention;

[0016]FIG. 2 is a functional block diagram showing the details of thedigital image processor shown in FIG. 1;

[0017]FIG. 3 is a function block diagram showing the details of thenoise estimation module shown in FIG. 2; and

[0018]FIG. 4 is a diagram showing the arrangement of the pixels in alocal region about a pixel of interest used to calculate the noise freepixel estimate.

DETAILED DESCRIPTION OF THE INVENTION

[0019] In the following description, a preferred embodiment of thepresent invention will be described as a software program. Those skilledin the art will readily recognize that the equivalent of such softwarecan also be constructed in hardware. Because image manipulationalgorithms and systems are well known, the present description will bedirected in particular to algorithms and systems forming part of, orcooperating more directly with, the method in accordance with thepresent invention. Other aspects of such algorithms and systems, andhardware and/or software for producing and otherwise processing theimage signals involved therewith, not specifically shown or describedherein can be selected from such systems, algorithms, components, andelements known in the art. Given the description as set forth in thefollowing specification, all software implementation thereof isconventional and within the ordinary skill in such arts.

[0020] The present invention can be implemented in computer hardware.Referring to FIG. 1, the following description relates to a digitalimaging system which includes an image capture device 10 a, an digitalimage processor 20, an image output device 30 a, and a general controlcomputer 40. The system can include a monitor device 50 such as acomputer console or paper printer. The system can also include an inputdevice control 60 for an operator such as a keyboard and or mousepointer. Multiple capture devices 10 a, 10 b, and 10 c are shownillustrating that the present invention can be used for digital imagesderived from a variety of imaging devices. For example, FIG. 1 canrepresent a digital photofinishing system where the image capture device10 a is a conventional photographic film camera for capturing a scene oncolor negative or reversal film combined with a film scanner device forsensing the developed image on the film and producing a digital image.Although the term “scanner” can refer to digital imaging devices thatphysically scan or move a sensing element past a photographic filmsample, the present invention also includes photographic film scannersand print scanners that employ a stationary image sensing device togenerate a digital image. The digital image processor 20 provides themeans for processing the digital images to produce pleasing lookingimages on the intended output device or media. Multiple image outputdevices 30 a and 30 b are shown illustrating that the present inventioncan be used in conjunction with a variety of output devices which caninclude a digital photographic printer and soft copy display. Thedigital image processor 20 processes the digital image to adjust theoverall brightness, tone scale, image structure etc. of the digitalimage in a manner such that a pleasing looking image is produced by animage output device 30 a. Those skilled in the art will recognize thatthe present invention is not limited to just these mentioned imageprocessing modules.

[0021] The general control computer 40 shown in FIG. 1 can store thepresent invention as a computer program stored in a computer readablestorage medium, which can include, for example: magnetic storage mediasuch as a magnetic disk (such as a floppy disk) or magnetic tape;optical storage media such as an optical disc, optical tape, or machinereadable bar code; solid state electronic storage devices such as randomaccess memory (RAM), or read only memory (ROM). The associated computerprogram implementation of the present invention can also be stored onany other physical device or medium employed to store a computer programindicated by offline memory device 70. Before describing the presentinvention, it facilitates understanding to note that the presentinvention is preferably utilized on any well-known computer system, suchas a personal computer.

[0022] It should also be noted that the present invention implemented ina combination of software and/or hardware is not limited to deviceswhich are physically connected and/or located within the same physicallocation. One or more of the devices illustrated in FIG. 1 can belocated remotely and can be connected via a wireless connection.

[0023] A digital image includes one or more digital image channels. Eachdigital image channel includes a two-dimensional array of pixels. Eachpixel value relates to the amount of light received by the image capturedevice 10 a corresponding to the geometrical domain of the pixel. Forcolor imaging applications a digital image will typically consist ofred, green, and blue digital image channels. Other configurations arealso practiced, e.g. cyan, magenta, and yellow digital image channels.Motion imaging applications can be thought of as a time sequence ofdigital images. Those skilled in the art will recognize that the presentinvention can be applied to, but is not limited to, a digital imagechannel for any of the above mentioned applications. Although thepresent invention describes a digital image channel as a two dimensionalarray of pixel values arranged by rows and columns, those skilled in theart will recognize that the methodology of the present invention can beapplied to mosaic (non rectilinear) arrays with equal effect.

[0024] The digital image processor 20 shown in FIG. 1 is illustrated inmore detail in FIG. 2. The general form of the digital image processor20 employed by the present invention is a cascaded chain of imageprocessing modules. A source digital image 101 is received by thedigital image processor 20 which produces on output an enhanced digitalimage 102 and a noise characteristic table 105, i.e. a table of noisecharacteristic values representing estimates of the noise magnitude forthe source digital image. A noise estimation module 110 receives thesource digital image 101 and produces the noise characteristic table105. Each image processing module contained within the digital imageprocessor 20 receives a digital image, modifies the digital image,produces a processed digital image and passes the processed digitalimage to the next image processing module. The two enhancement transformmodules shown within the digital image processor 20 are a noisereduction module 22 and a spatial sharpening module 23. These twomodules use the noise characteristic table 105 produced by the noiseestimation module 110 to produce the enhanced digital image 102. Thoseskilled in the art will recognize that any other image processing modulethat utilizes a noise characteristic table can be used with the presentinvention.

[0025] The noise estimation module 110 shown in FIG. 2 is illustrated inmore detail in FIG. 3. The source digital image 101 includes pixelscorresponding to one or more different colors and typically includesthree digital image channels that have pixels corresponding to red,green, and blue colors. The residual transform module 120 receives thesource digital image 101 and uses spatial filters and the pixel data ofthe source digital image 101 to calculate a residual digital image 107,i.e. a residual pixel value corresponding to each original pixel valuein the source digital image 101. Thus the residual digital image 107includes pixels having values corresponding to the one or more differentcolors of the source digital image 101. A residual statisticsaccumulator 130 receives the residual digital image 107 and calculates aset of residual histograms from the residual digital image 107. A noisetable calculator 140 receives the set of residual histograms andproduces a noise characteristic table 105.

[0026] The residual transform module 120 performs a spatial filteringoperation on the pixel data of the source digital image 101. That is, aresidual pixel value is generated for each selected pixel of interest inthe source digital image 101. In general, all or nearly all of thepixels of the source digital image 101 are selected as pixels ofinterest. However, it is important to note that the present inventioncan be practiced using a subset of the pixels of the source digitalimage 101 and still produce accurate noise characteristic tables. Foreach pixel of interest, a collection of pixel values sampled in a localregion about the pixel of interest is used to calculate two or morenoise-free pixel estimates for the pixel of interest. A final noise-freepixel estimate is chosen and the residual pixel value is calculated asthe difference between the final noise-free pixel estimate and the valueof the pixel of interest. The residual transform module 120 performs thespatial filtering operation on each color digital image channelindividually and forms a residual pixel value for each pixel of eachcolor digital image channel. That is, the spatial filtering operation ofthe red pixel values does not use the green pixel values and vice versa.The process is described mathematically below.

[0027] Let g(x,y) describe the array of pixel values corresponding to anindividual color digital image channel of the source digital image 101.Assuming the color digital image channel is affected by an additivenoise source, g(x,y) can be defined in terms of a noise component n(x,y)and a signal component f(x,y) given by (1).

g(x,y)=f(x,y)+n(x,y)  (1)

[0028] An estimate of the signal component f(x,y) is obtained using aspatial filter. The noise component n(x,y) is then obtained by computingthe difference between g(x,y) and the signal component f(x,y). Theeffectiveness of the overall noise estimation process depends largely onthe effectiveness of the spatial filter used. The better theapproximation of f(x,y), the better the estimate of the noise componentn(x,y). Ultimately, the goal is to produce a noise component (theresidual digital image 107), n(x,y), that is composed exclusively ofnoise. That is, there should be no image structure signal content in theresidual digital image 107.

[0029] The preferred embodiment of the present invention uses gradientanalysis on the pixels of the source digital image 101 to select themost suitable orientation for estimating noise for a pixel of interest.The gradient analysis is performed by applying four spatial filters eachof which intersects with the pixel of interest to obtain fourdirectional intensity gradient values in a local region sampled aboutthe pixel of interest. The four spatial filters represent a plurality oforientations for the pixel of interest since each has a differentimplied orientation with regard to edge detection. Specifically, thefour spatial filters provide a measure of the signal gradient in fourspatial directions oriented about the pixel of interest: 0, 90, 45, and135 degrees. The four spatial filters used to obtain the directionalintensity gradient values are given as: $\begin{matrix}{h_{0} = \begin{bmatrix}{- 1} & {- 2} & {- 1} \\0 & 0 & 0 \\1 & 2 & 1\end{bmatrix}} & (2) \\{h_{90} = \begin{bmatrix}{- 1} & 0 & 1 \\{- 2} & 0 & 2 \\{- 1} & 0 & 1\end{bmatrix}} & (3) \\{h_{45} = \begin{bmatrix}2 & 1 & 0 \\1 & 0 & {- 1} \\0 & {- 1} & {- 2}\end{bmatrix}} & (4) \\{h_{135} = \begin{bmatrix}0 & 1 & 2 \\{- 1} & 0 & 1 \\{- 2} & {- 1} & 0\end{bmatrix}} & (5)\end{matrix}$

[0030] The spatial filter of equation (2) produces a directionalintensity gradient value for the 0 degree direction, the spatial filterof equation (3) produces a directional intensity gradient value for the90 degree direction, the spatial filter of equation (4) produces adirectional intensity gradient value for the 45 degree direction, andthe spatial filter of equation (5) produces a directional intensitygradient value for the 135 degree direction. For a given pixel ofinterest g(x₀,_(y) ₀), the directional intensity gradient valuecorresponding to the 45 degree orientation, g′₄₅(x₀,y₀) is calculated byconvolving the spatial filter of equation (4) with the source digitalimage as: $\begin{matrix}{{g_{45}^{\prime}\left( {x_{0},y_{0}} \right)} = {\sum\limits_{i = {- 1}}^{1}\quad {\sum\limits_{j = {- 1}}^{1}\quad {{g\left( {{x_{0} - i},{y_{0} - j}} \right)}{h_{45}\left( {i,j} \right)}}}}} & (6)\end{matrix}$

[0031] where g′₄₅(x₀,y₀) is the estimate of the intensity gradient inthe 45 degree orientation at pixel of interest g(x₀,y₀). Similarly, thedirectional intensity gradient values for the other spatial orientationsare calculated by substituting the corresponding spatial filter for theh45(i,j) term of equation (6). The spatial filters described inequations (2) through (5) are the well-known, Sobel gradient filters.Those skilled in the art will recognize that other analogous gradientdetecting spatial filters exist and can be used to calculate thedirectional intensity gradient values. For instance the following twosimple spatial filters could be used instead of those shown in equations(2)-(5):

h₉₀=[−1 0 1]  (7)

h₀ =h ₉₀ ^(T)  (8)

[0032] where the spatial filter, h₀, is given as the transpose of h₉₀,as shown in equation (8). The spatial filters shown in equations (7)-(8)are much simpler to implement and represent spatial filters that can beused to calculate two directional intensity gradient values for thespatial orientation directions 0 and 90 degrees. Another example of aset of spatial filters that can be used are the Sobel spatial filtersthat are the same as described by (2), (3), (47), and (5) with theexception that elements with values of 2.0 are replaced with a value of1.0.

[0033] An important aspect of the present invention is the use ofmultiple directional intensity gradient values calculated for differentspatial directions. The underlying image structure, represented byf(x,y), very often exhibits spatial correlation along a particulardirection. The goal is to obtain the most robust estimate of f(x,y) andthus eliminate the noise. Similarly, for noise estimation purposes, arobust estimate of f(x,y) ensures that n(x,y) contains no residualspatial structure from f(x,y). In other words, the key is todifferentiate the noise from the spatial structure. Given that noise isgenerally of high spatial frequency, noise can be easily differentiatedfrom spatial structure when the spatial structure is of low spatialfrequency. Experimentation with natural images has shown that the localgradient (i.e. in the vicinity of a pixel of interest) can be a goodindication of the spatial frequency of the underlying spatial structure.In particular, the magnitude of a directional intensity gradient valuecan be used as in indication of the orientation of the underlyingspatial structure. Therefore, the directional intensity gradient valueof minimum magnitude, i.e. the minimum absolute value, indicates thedirection for which the spatial structure is of lowest spatialfrequency. Thus a robust noise estimate is obtained by selecting thepreferred spatial orientation based on the minimum magnitude directionalintensity gradient value.

[0034] It should also be noted that in the above mentioned gradientspatial filters that the value of the pixel of interest is not used inthe calculation of the directional intensity gradient values. Therefore,each directional intensity gradient value is independent from the valueof the pixel of interest.

[0035] Having applied the spatial filters of equations (2)-(5) to theimage pixel data, we now have four directional intensity gradient valuescalculated for the pixel of interest. We will then use the directionalintensity gradient values to select a preferred spatial orientation forthe pixel of interest. The selected preferred spatial orientation isthen used to calculate a noise-free pixel estimate for the pixel ofinterest relating to the signal f(x,y). The noise-free pixel estimatefor the pixel of interest is calculated using the values of pixelssampled about the pixel of interest. Experimentation with digital imagedata has shown that the accuracy of the noise estimation process isimproved for image regions characterized by low signal modulation.Therefore, the present invention calculates the noise-free pixelestimate by preferentially weighting the pixels along the directionindicated by the preferred spatial orientation, i.e. the directioncorresponding to the lowest spatial frequency modulation. Using the fourdirectional intensity gradient values, we use the following logic toobtain the noise-free pixel estimate:

[0036] If max {g′₀(x₀,y₀), g′₉₀(x₀,y₀), g′₄₅(x₀,y₀),g′₁₃₅(x₀,y₀)}=g′₀(x₀,y₀),

[0037] then${f\left( {x_{0},y_{0}} \right)} = {\frac{1}{3}\left( {A_{4} + A_{5} + A_{6}} \right)}$

[0038] Else, if max {g′₀(x₀,y₀), g′₉₀(x₀,y₀), g′₄₅(x₀,y₀),g′₁₃₅(x₀,y₀)}=g′₉₀(x₀,y₀),

[0039] then${f\left( {x_{0},y_{0}} \right)} = {\frac{1}{3}\left( {A_{2} + A_{5} + A_{8}} \right)}$

[0040] Else, max {g′₀(x₀,y₀), g′₉₀(x₀,y₀), g′₄₅(x₀,y₀),g′₁₃₅(x₀,y₀)}=g′₄₅(x₀,y₀),

[0041] then${f\left( {x_{0},y_{0}} \right)} = {\frac{1}{3}\left( {A_{3} + A_{5} + A_{7}} \right)}$

[0042] Else, max {g′₀(x₀,y₀), g′₉₀(x₀,y₀), g′₄₅(x₀,y₀),g′₁₃₅(x₀,y₀)}=g′₄₅(x₀,y₀), then${f\left( {x_{0},y_{0}} \right)} = {\frac{1}{3}\left( {A_{1} + A_{5} + A_{9}} \right)}$

[0043] where the values A₁ through A₉ shown above are pixels in a localregion sampled about the pixel of interest and f(x₀,y₀) is thenoise-free pixel estimate for the pixel of interest located at imagecoordinates (x₀,y₀). The local region used to obtain the noise-freepixel estimate is shown in more detail in FIG. 4. ZZZ As can be seenfrom equations (2)-(5) and FIG. 4, the preferred embodiment of thepresent invention uses a 3×3 local region sampled about the pixel ofinterest. It should be noted, however, that a different sized localregion can be used. For instance, a 5×5 local region might be desirable.Those skilled in the art will note that our proposed framework is easilyextended to any such case.

[0044] The spatial filtering technique described above can be used fornoise estimation and noise removal. As described hereinbelow, the noisefree pixel estimate is subtracted from the pixel of interest to form anoise residual image from which an estimate of the noise content can bederived. The present invention also uses the spatial filtering techniqueto form an enhanced digital image from the noise free pixel estimatevalues.

[0045] Having obtained the noise-free estimate f(x,y) for the pixel ofinterest, an estimate of the noise n(x,y) can obtained by rearrangingequation (1) as:

n(x,y)=g(x,y)−f(x,y)  (9)

[0046] Thus the noise residual image 107 is obtained by subtracting thenoise free pixel estimates from the values of the corresponding pixelsof interest.

[0047] As described above, the pixel data of the source digital image101 can be conceptualized as having two components—a signal componentrelating to photographed objects f(x,y) and a noise component n(x,y).The resulting residual pixel values have statistical properties thathave a closer relationship to the noise component of the pixel data ofthe source digital image 101 than the signal component. Although thenoise component can contain sub-components, the stochastic sub-componentof the noise component is well modeled by a zero mean Gaussianprobability distribution function. To first order, the noise componentof the pixel data of the source digital image 101 can be characterizedby a standard deviation and a mean value of zero. To second order,standard deviation of the noise component can be modeled as being signalstrength and color dependent.

[0048] Referring to FIG. 3, a residual statistics accumulator 130analyzes the residual pixel values and records these values in the formof a set of residual histograms as a function of the color digital imagechannel and pixel value. Therefore a given residual histogram H_(ik)relates to the i^(th) color digital image channel and the k^(th) pixelvalue sub-range. For each pixel of interest denoted byp_(mn)(corresponding to the m^(th) row and n^(th) column location) inthe processed color digital image channel, a histogram bin index k iscomputed. For example, if the numerical range pixel values is from 0 to255 there can be as many as 256 useful histograms, i.e. one histogramfor each possible numerical pixel value. In general, most noise sourcescan be characterized as having noise standard deviations that are slowfunctions of the pixel value. Therefore, the preferred embodiment of thepresent invention uses 8 histograms to cover the numerical pixel valuerange of 0 to 255. Thus the calculated histogram index bin and thecorresponding sub-range pixel values are given by the following Table(1). TABLE 1 histogram bin index sub-range pixel values average pixelvalue 0  0 to 31 16 1 32 to 63 48 2 64 to 95 80 3  96 to 127 112 4 128to 159 144 5 160 to 191 176 6 192 to 233 208 7 234 to 255 240

[0049] Those skilled in the art will recognize that the presentinvention can be practiced with digital image pixel data with anynumerical range. The number of residual histograms used for each colordigital image channel will depend on the accuracy of results requiredfor the particular digital imaging application.

[0050] Although each approximate residual histogram records statisticalinformation for a range of pixel values for a given color digital imagechannel, the residual histogram records the frequency of residual pixelvalues associated with each pixel of interest p_(mn). Since the expectedmean of the distribution of residual pixel values is zero, the residualpixel values exhibit both positive and negative values. Therefore, theapproximate residual histogram must record the frequency, i.e. thenumber of instances of residual pixel values, of all possible instancesof residual pixel values. For the example above, the residual pixelvalues can range from −255 to +255. While is possible to constructresidual histograms with as many recording bins as there are possibleinstances of residual pixel values, in general it is not necessary. Formost digital images only a small percentage of residual pixel valuesexhibit values near the extremes of the possible range. The presentinvention uses 101 total recording bins for each residual histogram. Onof the recording bins corresponds to residual pixel values of 50 andgreater. Similarly, one other recording bin corresponds to residualpixel values of −50 and lower. The other 99 recording bins eachcorrespond to a single residual pixel value for the numerical range from−49 to +49.

[0051] Referring to FIG. 3, the noise table calculator 140 receives theset of residual histograms and calculates a noise characteristic table.For each of the residual histograms relating to a particular colordigital image channel and pixel value range, the noise table calculator140 derives a noise standard deviation value from the value of therecording cells of the updated residual histogram. This is equivalent tocalculating the standard deviation of the noise residual pixel values.The preferred embodiment of the present invention uses equation (10) tocalculate the standard deviation value σ_(n)

σ_(n)=((1/N)Σ_(k) RC _(v)(k)(x−x _(m))²)^(1/2)  (10)

[0052] where the variable x represents the average pixel value of theresidual pixel values accumulated in the k^(th) recording cell as givenby Table (1) and RCv(k) represents the number of residual pixel valuesaccumulated by the k^(th) recording cell.

x=V(k)  (11)

[0053] The variable x_(m) represents the arithmetic mean value of thecorresponding residual pixel values given by equation (9) and,

x _(m)=(1/N)Σ_(k) x  (12)

[0054] and the variable N represents the total number of residual pixelvalues recorded by the updated residual histogram given by equation(13).

N=Σ _(k) RC _(v)(k)  (13)

[0055] An alternative embodiment of the present invention performs analpha-trimmed standard deviation calculation. In this embodiment a firstapproximation to the standard deviation σ_(e) is calculated using themethod described above. The calculation of σ_(n) is then calculatedusing the only recording cells with corresponding residual pixel valuesthat are within a limited range of zero. The formula for the standarddeviation calculation σ_(n) is given by equation (14)

σ_(n)=((1/N)Σ_(k) γRC _(v)(k)(x−x _(m))²)1/2  (14)

[0056] where the variable γ is given by equation (15)

γ=1 if |x|<ασ_(e)

γ=0 if |x|>=ασ_(e)  (15)

[0057] where the variable α is set to 3.0. This alternative embodimentof the present invention is more computationally intensive than thepreferred embodiment but does yield more accurate results via therejection of out-lying residual pixel values from adversely contributingto the calculation of the standard deviation σ_(n) value.

[0058] Table 2 below is an example of a noise characteristic tablepresent invention. TABLE 2 standard standard standard average pixeldeviation of deviation of deviation of valve red channel green channelblue channel 16 1.775 1.828 1.899 48 1.767 1.820 2.231 80 1.990 2.0752.136 112 2.100 2.151 1.888 144 1.703 2.093 2.058 176 0.885 0.923 2.356208 0.937 0.917 0.977 240 1.279 1.660 0.928

[0059] Those skilled in the art should recognize that the presentinvention can be practiced with calculated quantities other than thestandard deviation that relate to the noise present in digital images.For example, the statistical variance or statistical median can also bederived from the residual histograms and be used to form a table ofnoise characteristic values. As can be seen from Table 2, the noisecharacteristic value is reported as a function of the numerical valuesof the source digital image pixels—i.e. the light intensity valuesrepresented in the source digital image. In addition, if the sourcedigital image contains two or more color channels, the noisecharacteristic values can also be calculated as a function of color andthe numerical values of the source digital image pixels (as is the casein Table 2).

[0060] The accuracy of the noise characteristic value estimates yieldedby the present invention can be improved with additional refinement ofthe residual pixel values. For example, if the source digital imagecontains two or more color channels, the residual pixel values for thetwo or more color channels can be used to calculate a color weightingfactor. This color weighting factor can then be used to exclude residualpixel values from the calculation of the noise characteristic value.Those skilled in the art will note that this type of refinement is welldocumented in the literature and could easily be combined with thepresent invention.

[0061] The present invention uses a set of residual histograms to recordthe calculated statistics. A set of histograms is an example of astatistical table from which a noise characteristic table can bederived. Thus the set of residual histograms constitutes a statisticaltable. Those skilled in the art should recognize that the presentinvention can be practiced with other forms of statistical tables. Forexample, the residual digital images can be stored and serve as astatistical table.

[0062] The calculated noise characteristic table is used in conjunctionwith spatial filters for the purpose of enhancing the source digitalimage 101 and thus produce an enhanced digital image 102. A spatialfilter is any method which uses pixel values sampled from a local regionabout a pixel of interest to calculate an enhanced pixel value whichreplaces the pixel of interest. Those spatial filters which reducespatial modulation, for at least some pixels in an effort to removenoise from the processed digital image, can be considered noisereduction filters. Those spatial filters which increase spatialmodulation, for at least some pixels in an effort to enhance spatialdetail noise in the processed digital image, can be considered spatialsharpening filters. It should be noted that it is possible for a singlespatial filter to be considered both a noise reduction filter as well asa spatial sharpening filter. The present invention can be used with anydigital image processing method which makes uses of a noisecharacteristic table to produce an enhanced digital image 102. Spatialfilters that adjust a processing control parameter as a function ofeither the color or numerical value of pixels are adaptive spatialfilters. The present invention uses a noise reduction filter and aspatial sharpening filter which are responsive to a noise characteristictable.

[0063] Referring to FIG. 2, the preferred embodiment of the presentinvention employs a noise reduction module 22 as part of the imageprocessing method to produce enhanced digital images 102. As such, thesource digital image 101 and the noise characteristic table 105 arereceived by the noise reduction module 22 which produces on output anoise reduced digital image.

[0064] It is important to note that for many practical digital imagingimage systems, other image processing processors need to be included. Aslong as these other image processing processors accept a digital imageas input and produce a digital image on output, more of these type ofimage processing modules can be inserted in the image processing chainin between a noise reduction module 22 and a spatial sharpening module23.

[0065] The present invention uses a modified implementation of the Sigmafilter, described by Jong-Sen Lee in the journal article Digital ImageSmoothing and the Sigma Filter, Computer Vision, Graphics, and ImageProcessing Vol 24, p. 255-269, 1983, as a noise reduction filter toenhance the appearance of the processed digital image. The values of thepixels contained in a sampled local region, n by n pixels where ndenotes the length of pixels in either the row or column direction, arecompared with the value of the center pixel, or pixel of interest. Eachpixel in the sampled local region is given a weighting factor of one orzero based on the absolute difference between the value of the pixel ofinterest and the local region pixel value. If the absolute value of thepixel value difference is less or equal to a threshold ε, the weightingfactor if set to one. Otherwise, the weighting factor is set to zero.The numerical constant ε is set to two times the expected noise standarddeviation. Mathematically the expression for the calculation of thenoise reduced pixel value is given as

q _(mn)=Σ_(ij) a _(ij) p _(ij)/Σ_(ij) a _(ij)

and

a_(ij)=1 if |p_(ij) −p _(mn)|<=ε

a_(ij)=0 if |p _(ij) −p _(mn)|>ε  (16)

[0066] where p_(ij) represents the ij^(th) pixel contained in thesampled local region, p_(mn) represents the value of the pixel ofinterest located at row m and column n, a_(ij) represents a weightingfactor, and q_(mn) represents the noise reduced pixel value. Typically,a rectangular sampling region centered about the center pixel is usedwith the indices i and j varied to sample the local pixel values.

[0067] The signal dependent noise feature is incorporated into theexpression for ε given by equation (17)

ε=Sfacσ _(n)(p_(mn))  (17)

[0068] where σ_(n) represents the noise standard deviation of the sourcedigital image evaluated at the center pixel value p_(mn) as described byequations (10) and (14) above. The parameter Sfac is termed a scalefactor can be used to vary the degree of noise reduction. The optimalvalue for the Sfac parameter has been found to be 1.5 throughexperimentation however values ranging from 1.0 to 3.0 can also produceacceptable results. The calculation of the noise reduced pixel valueq_(mn) as the division of the two sums is then calculated. The processis completed for some or all of the pixels contained in the digitalimage channel and for some or all the digital image channels containedin the digital image. The noise reduced pixel values constitute thenoise reduced digital image. The modified implementation of the Sigmafilter is an example of a noise reduction filter that uses a noisecharacteristic table and is therefore an adaptive noise reduction filterwhich varies the amount of noise removed as a function of the pixelcolor and numerical value.

[0069] Referring to FIG. 2, the preferred embodiment of the presentinvention employs a spatial sharpening module 23 as part of the imageprocessing method to produce an enhanced digital image 102. As such, thenoise reduced digital image and the noise characteristic table 105 arereceived by the spatial sharpening module 23 which produces on output anenhanced digital image 102.

[0070] Although the present invention can be used with any spatialsharpening filter which utilizes a priori knowledge of the noisecharacteristics to sharpen the spatial detail of the processed digitalimage. The preferred embodiment uses a modified implementation of themethod described by Kwon et al in U.S. Pat. No. 5,081,692. This spatialsharpening method performs an unsharp masking operation by filtering theinput digital image with a spatial averaging 2-dimensional Gaussianfilter (characterized by a standard deviation of 2.0 pixels) whichresults in a blurred digital image. The blurred digital image issubtracted from the input digital image to form a high-pass residual. Inthe method disclosed by Kwon et al. a local variance about a pixel ofinterest is calculated by using the pixel data from the high-passresidual. Based on the value of the local variance a sharpening factoris adjusted so as to amplify large signals more than small amplitudesignals. The amplification factor Φ is therefore a factor of the localvariance v. i.e. Φ (v).

[0071] The present invention modifies the method taught by Kwon et al.to make the amplification factor Φ (v) a function of the estimatednoise, i.e. Φ(v,σ_(n)). The amplification function f is given by a gammafunction, or integral of a Gaussian probability function, as given byequation (18). $\begin{matrix}{{\varphi (v)} = \frac{{y\quad o} + {y\quad \max \quad S\quad ɛ^{{- {({\varpi - \varpi_{o}})}}{2/2}\quad {\sigma 2}}}}{{\psi \quad o} + {\psi \quad \mu \quad \alpha \quad \xi \quad ɛ^{{- {({\varpi_{\mu \quad \alpha \quad \xi} - \varpi_{o}})}}{2/2}\quad {\sigma 2}}}}} & (18)\end{matrix}$

[0072] where y_(o) represents a minimum amplification factor y_(max)represents a maximum amplification factor, v_(max), represents a maximumabscissa value of the variable v, v_(o), represents a transitionparameter and s represents a transition rate parameter. The variablev_(o) is a function of the noise standard deviation value σ_(n) as perequation (19)

v _(o) =Sfac ₂σ_(n)(p_(mn))  (19)

[0073] where the scaling factor Sfac₂ determines the sensitivity of thesharpening sensitivity to the noise and the noise standard deviationvalue σ_(n) is as described above in equations (10) and (14). Theoptimal values for the variables used in equation (19) depend on thedigital imaging application. The present invention uses a value of 1.0for y_(o) which results in no spatial sharpening for noisy regions. Avalue of 3.0 is used for y_(max), however, this variable is sensitive touser preference with values ranging from 2.0 to 4.0 producing acceptableresults. The value of Sfac₂ should be set to between 1.0 and 2.0 with1.5 as optimal. The variables should be set to values in the range fromv_(o)/2 to v_(o)/10 for reasonable results. The variable v_(max), shouldbe set to a value much larger than the expected noise, e.g. 20 time thevalue of σ_(n).

[0074] While the preferred embodiment of the present inventioncalculates a noise characteristic table and then subsequently uses thenoise characteristic table to produce an enhanced digital image, somedigital imaging systems can be configured to separate the calculationphase from the enhancement phase. In an alternative embodiment of thepresent invention, the calculated noise characteristic table is storedwith the source digital image 101 as meta-data, i.e. non-pixelinformation. The source digital image 101 with meta-data can betransmitted to a remote site or stored for safe keeping to be used at alater time or another site. Any of the above mentioned noisecharacteristic tables can be stored as meta-data. In general a noisecharacteristic table requires much less memory storage than a set ofresidual histograms. However, a set of residual histograms can be storedwith the source digital image 101 as meta-data.

[0075] The present invention uses a spatial filter to calculate aresidual digital image 107 from a source digital image 101 and derivesnoise characteristic values from the residual digital image 107. Thoseskilled in the art will recognize that the present invention can be usedin conjunction with spatial masking techniques, such as the methoddescribed by Snyder et al. in commonly-assigned U.S. Pat. No. 5,923,775,to improve the statistical accuracy of the method.

[0076] The four direction spatial filter described above can be used asa noise reduction filter. In this embodiment of the present invention,the final noise free pixel estimates are calculated for each pixel inthe source digital image 101. The final noise free pixel estimatestherefore forms a noise reduced digital image, i.e. a representation ofthe source digital image 101 with noise removed. An advantage of thepresent invention over other noise reduction method is the fact that thepresent invention does not require a priori knowledge of the noisecharacteristics of the source digital image 101.

[0077] The invention has been described in detail with particularreference to certain preferred embodiments thereof, but it will beunderstood that variations and modifications can be effected within thespirit and scope of the invention.

Parts List

[0078]10 a image capture device

[0079]10 b image capture device

[0080]10 c image capture device

[0081]20 digital image processor

[0082]22 noise reduction module

[0083]23 spatial sharpening module

[0084]30 a image output device

[0085]30 b image output device

[0086]40 general control computer

[0087]50 monitor device

[0088]60 input control device

[0089]70 offline memory device

[0090]101 source digital image

[0091]102 enhanced digital image

[0092]105 noise characteristic table

[0093]107 residual digital image

[0094]110 noise estimation module

[0095]120 residual transform module

[0096]130 residual statistic accumulator

[0097]140 noise table calculator

What is claimed is:
 1. A method of estimating the noise in a digitalimage for use in subsequent image processing, comprising the steps of:a) receiving a source digital image having a plurality of pixels; b)selecting a pixel of interest; c) providing a plurality of orientationsfor the pixel of interest; d) using gradient analysis on the sourcedigital image and the plurality of orientations to select the mostsuitable orientation for estimating noise for the pixel of interest; e)using the selected orientation in the pixel of interest to determine anoise-free pixel estimate for the pixel of interest; and f) repeatingsteps b)-e) for other pixels of interest and using the noise-free pixelestimates to calculate a noise characteristic value representing thenoise estimate for the source digital image.
 2. The method of claim 1wherein step d) includes calculating two or more directional intensitygradient values for the pixel of interest wherein each directionalintensity gradient value corresponds to a different spatial orientationabout the pixel of interest.
 3. The method of claim 1 wherein step e)includes calculating a residual pixel value using the noise-free pixelestimate and the value of the pixel of interest.
 4. The method of claim3 wherein step f) includes using the residual pixel values to calculatethe noise characteristic value.
 5. The method of claim 2 wherein eachdirectional intensity gradient value is independent from the value ofthe pixel of interest.
 6. The method of claim 4 further includingcalculating the residual pixel value as the difference between thenoise-free pixel estimate and the value of the pixel of interest.
 7. Themethod of claim 2 wherein the directional intensity gradient value ofminimum value is used to select the most suitable orientation for thepixel of interest.
 8. The method of claim 2 wherein Sobel spatialfilters are used to calculate the directional intensity gradient values.9. The method of claim 2 wherein Prewitt spatial filters are used tocalculate the directional intensity gradient values.
 10. The method ofclaim 1 wherein step f) includes calculating the noise characteristicvalue as a function of the numerical values of the source digital imagepixels.
 11. The method of claim 1 wherein step f) includes calculatingthe noise characteristic value as a function of color and the numericalvalues of the source digital image pixels.
 12. The method of claim 3wherein step f) includes calculating the noise characteristic value asthe standard deviation of the noise residual pixel values.
 13. Themethod of claim 1 further includes using the noise characteristic valueto remove noise from the source digital image to produce an enhanceddigital image.
 14. The method of claim 1 further includes using thenoise characteristic value to sharpen the spatial detail of the sourcedigital image to produce an enhanced digital image.
 15. The method ofclaim 1 wherein each of the plurality of orientations for the pixel ofinterest intersect with the pixel of interest.